I would also like to add to my other comment this experience :
When I was still active I was friends with a woman that was inactive .Several of the Elders wives would make snide remarks about this or ask personal questions about my friend . I would always tell them to call and ask her themselves .... Then one day one of my best friends, who was also an Elders wife ,told me that if I did not cut off association with the inactive sister ,then she would no longer be able to be my friend .She used the same logic as they do with DFd persons " You have to cut off association so that they will see what they are missing ".She also said if this sister is involved with worldly things than I am guilty too by association.
So don't tell me you can just walk away !